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1.  Introduction 


For  today's  large,  software-intensive  systems,  the  length  of  the  development  cycle  and  the 
number  and  complexity  of  technical  and  organizational  interfaces  create  a  great  deal  of  uncertainty 
and  risk.  Additionally,  for  many  of  these  systems,  the  government's  acquisition  philosophy  dictates 
that  minimal  standards  and  contractor  controls  be  included  in  the  contract,  which  results  in  the 
government  having  little  insight  into  the  quality  of  the  developing  software-intensive  product.  It  is, 
therefore,  necessary  to  be  able  to  objectively  evaluate  these  systems  during  their  development  to 
determine  whether  or  not  they  will  meet  requirements,  schedule,  and  budget;  to  assist  risk 
management;  and  to  facilitate  corrective  and  preventive  action.  Software  system  metrics  can  provide 
objective  information  necessary  for  technical  and  managerial  insight  into,  control  of,  and 
improvement  of  the  development  effort. 

Over  the  last  few  years.  Computer  Systems  Division  personnel  have  developed  a  metrics 
approach  that  has  been  designed  for  use  during  the  development  of  large  software-intensive  systems. 
This  approach  includes  an  integrated  set  of  system-  and  software-level  metrics  recommended  for 
collection  by  the  development  contractor(s)  and  detailed  descriptions  of  each  of  these  metrics.  In 
creating  this  set  of  recommended  metrics  and  their  descriptions,  the  results  of  other  current  metrics 
technology  efforts  were  incorporated,  as  appropriate.  The  metrics  approach  also  includes  suggested 
tailorings  to  selected  contractual  documentation  to  ensure  that  needed  metrics  information  will  be 
collected  and  reported  to  the  Government.  The  metrics  approach,  recommended  contractual 
documentation  guidelines,  and  detailed  descriptions  of  several  of  the  recommended  metrics  are  the 
subject  of  an  unpublished  TOR,  "Metrics  Guidelines  for  Software  Intensive  MCCR  Systems." 
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2.  Metrics  Within  Large  Systems:  Three  Key  Concepts 


Three  basic  concepts  recommended  for  the  development  of  large  software-intensive  systems 
include  seamlessness,  consistency,  and  defined  expectations.  These  concepts  apply  to  many  aspects  of 
development,  such  as  supportability  and  reliability,  as  well  as  to  metrics.  The  seamless  concept 
recognizes  that  most  of  our  software  systems  will  be  developed  by  a  prime  and  several  subcontractors. 
This  means  that  all  software  system  products  should  be  consistent  among  all  contractors  so  that  the 
exact  identity  of  the  developer  (i.e.,  prime  or  subcontractors)  is  transparent  to  the  Government.  For 
similar  software,  uniform  methods  and  types  of  tools  and  uniform  training  in  these  methods  and  tools 
are  recommended.  Thus,  in  accordance  with  the  concept  of  seamlessness,  all  contractors  should 
collect  and  report  the  same  metrics  information  so  that  a  uniform  set  of  metrics  information  is 
reported  to  the  program  office. 

The  consistency  concept  recognizes  that  the  total  software  process  is  an  integral  part  of  the 
overall  systems  engineering  process  and  must  be  dealt  with  as  such  throughout  the  entire  life  cycle 
and  across  all  systems  engineering  disciplines.  The  systems  engineering  process  has  a  system-level 
component  to  the  process,  which  then  flows  down  to  hardware-  and  software-level  subprocesses. 
Consistency  among  these  levels  is  necessary.  For  a  large  system,  metrics  should  be  collected  at 
several  levels:  system,  segment,  and  lower  levels.  Within  the  lower  levels,  there  are  hardware-  and 
software-specific  components.  The  software-level  metrics  program  has  been  created  to  be  consistent 
with  and  provide  information  to  the  higher  level  measurements.  The  higher  level  process  will  detail 
the  methods  by  which  lower  level  measures  are  incorporated  into  higher  level  measures.  Software- 
level  metrics  are  defined  to  be  those  that  deal  with  software-only  components;  integrated 
hardware/software  components  are  handled  by  higher  level  measurements. 

The  metrics  approach  includes  effective,  early  communication  of  Government  technical 
expectations  to  the  contractor(s)  before  Engineering  and  Manufacturing  Development  (EMD)  so  that 
the  contractor(s)  can  create  appropriate  plans  to  meet  these  expectations.  It  is  recommended  that  this 
be  done  by:  delivering  Government  expectations  documents  to  the  Dem/Val  contractors  before  they 
begin  developing  their  EMD  planning  documentation;  participating  in  Government-contractor 
Integrated  Product  Teams;  and  providing  feedback  on  early  versions  of  developing  planning 
documentation.  One  purpose  of  the  TOR  referenced  in  Section  1  is  to  provide  such  metrics 
expectations  to  the  contractor. 
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3.  Metrics  Planning:  Basic  Guidelines 


3.1  Primitive  and  Aggregate  Measures 

The  purpose  of  the  metrics  program  is  twofold:  to  gain  visibility  into  the  overall  health  and 
status  of  the  evolving  software  system  and  to  identify  at  the  earliest  possible  point  in  the  life  cycle, 
specific  problem  areas  or  potential  future  problems.  Both  detailed  and  aggregate  measures  are 
necessary  and  need  to  be  reported  to  the  Government  on  a  regular  basis  (often  monthly).  To  assess 
overall  health  and  status,  cumulative  measures  should  generally  be  used,  whereas  for  the  identification 
and  resolution  of  problems,  metrics  should  be  reported  at  a  detailed  level.  Detailed  or  primitive 
information  should  be  reported  (or  made  available)  in  electronic  form  for  analysis  and  retention  by 
the  program  office. 

3.2.  Metrics  Descriptions 

Emphasis  is  placed  on  the  need  for  careful  definition  and  description  of  each  metric  and  its 
report  formats.  Without  specific  definitions  of  precisely  what  is  being  measured,  the  measurement 
will  have  little  meaning  or  use.  It  is,  for  example,  insufficient  to  report  source  lines  of  code  (SLOC), 
without  discussing  how  that  code  is  being  counted.  A  definition  that  excludes  data  declarations  and 
comments  and  counts  only  executable  SLOC  may  easily  result  in  a  metric  value  that  is  half  that 
resulting  from  a  definition  that  includes  data  declarations  and  comments.  Additionally,  without 
relatively  consistent  descriptions  of  a  given  metric  that  is  used  on  several  different  programs,  it  will 
not  be  possible  to  adequately  evaluate  the  usefulness  of  reported  metric  data. 

3.3  Metrics  Collection/Reporting  Tools 

It  is  expected  that  whenever  possible,  the  collection  of  metrics  data  will  be  automated  and  will 
use  tools  that  have  been  integrated  into  the  contractor's  software  engineering  environment.  In 
general,  it  is  preferable  to  use  commercial  tools  when  they  are  available.  However,  for  some  metrics  it 
may  be  necessary  to  use  contractor-developed  tools,  either  because  there  are  no  commercial  tools  that 
calculate  the  defined  metric  or  because  the  contractor  tool  already  supports  some  aspect  of  the 
existing  development  process  and  that  aspect  is  being  measured.  For  example,  if  the  contractor  has 
an  existing  automated  problem  report  tracking  tool,  then  accumulating  metrics  on  problem  reports 
may  be  done  most  efficiently  by  modifying  the  existing  tool  to  collect  the  defined  metric.  The  same 
metrics  tools  should  be  used  by  all  development  contractors,  and  to  the  extent  possible,  all  tools  and 
methods  should  be  compatible  and  integrated  among  all  levels  of  the  software  system. 
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3.4  Contractor  Metrics  Plans 

The  contractor's  process  planning  documentation  (systems  and  software  level)  should  include 
a  detailed  and  unambiguous  definition  of  each  metric  and  its  report  formats,  or  should  reference 
Government-provided  definitions  and  report  formats  that  the  contractor  intends  to  use.  The  plans 
should  also  include  descriptions  of  methods/tools  used  to  collect,  analyze,  and  report  metric 
information,  as  well  as  a  description  of  management's  use  of  the  collected  metric  information  to  assess 
and  improve  the  software  system  product  and  the  processes  used  to  generate  the  product. 

3.5  Metrics  for  Many  Disciplines 

For  software,  the  metrics  program  is  designed  to  share  information  with  many  software 
disciplines  (e.g.,  risk  management.  Software  Quality  Assurance,  testing,  management,  and  problem 
reporting).  The  contractor's  software  planning  document  should  discuss  the  various  software 
organizations/activities  that  use  metric  data.  The  use  of  metric  information  to  assess  software  risk,  to 
assess  and  improve  software  processes,  to  manage  the  technical  effort,  and  to  identify  error-prone 
software  units  should,  for  example,  be  explained. 
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4.  Recommended  Metrics 


The  activities  of  selecting  and  defining  a  set  of  metrics  that  effectively  covers  the  software 
process  can  only  reach  closure  in  the  context  of  the  specific  development  processes  to  be  used. 
However,  it  is  possible  to  list  a  general  set  of  software  metrics  which  covers  the  main  activities  and 
phases  of  the  software  life  cycle.  This  set  can  be  tailored  and  specific  metric  definitions  can  be 
developed  to  suit  a  specific  software  life  cycle  and  process. 

Table  1  shows  an  example  set  of  metrics  that  covers  the  software  life  cycle.  Three  categories 
of  metrics  have  been  identified:  progress,  resource,  and  product/process.  A  collection  of  metrics 
from  each  of  these  categories  is  usually  required  for  comprehensive  coverage.  Progress  metrics 
indicate  an  organization's  adherence  to  schedule.  Resource  metrics  indicate  the  amount  of 
development,  integration,  test,  and/or  support  resources  and  personnel  available  and  the  amount  in 
use.  Product/process  metrics  are  used  to  measure  attributes  of  the  documentation  (electronic  and/or 
paper)  and  code  and  characteristics  of  the  activities,  methods,  practices,  and  transformations 
employed  in  developing  the  products.  Product  and  process  measurement  activities  tend  to  overlap, 
which  is  why  they  are  combined  into  one  category.  For  example,  a  high  number  of  product  defects 
can  imply  the  existence  of  a  problem  in  the  process  used  to  create  the  product.  Also,  a  dearth  of 
exposed  defects  can  indicate  the  existence  of  a  superior  product  or  a  deficient  inspection  process. 

While  it  is  necessary  to  have  a  software  metric  set  that  spans  the  software  life  cycle  and  is 
tailored  to  the  process,  this  is  not  sufficient  for  a  software  effort  that  will  be  integrated  into  a  larger 
system.  Thus,  we  also  recommend  use  of  a  set  of  progress  and  product/process  metrics  at  the  system 
level  that  is  integrated  and  consistent  with  the  software-level  metric  set,  and  these  metrics  are  listed  in 
Table  2.  Summary  descriptions  of  each  type  of  metric  listed  in  Tables  1  and  2  appear  in  Tables  3 
through  5. 
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Table  1.  Recommended  Metrics  for  the  Software  Measurement  Program 


'Definition  complete  or  in  progress. 
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Table  2.  Recommended  Metrics  for  the  System  Measurement  Program 


PROGRESS 


Requirements  Progress 
•  Specification  Completeness* 

-  System/Segment 

-  Integrated  (Hardware  and  Software)  Configuration  Item  (Cl) 

-  Hardware  Configuration  Item  (HWCI) 


Design  Progress 

•  System/Segment  Design  Document  Completeness 

•  Design  Completeness 

-  System/Segment 

-  Integrated  Cl 
-HWCI 

Integration  and  Test  Progress 

•  Test  Documentation  Completeness 

•  Test  Completeness 

-  System/Segment 

-  Integrated  Cl 
-HWCI 


PRODUCT  AND  PROCESS 


Volatility 

•  Requirements* 

-  System/Segment 

-  Integrated  Cl 
-HWCI 

•  Design 

-  System/Segment 

-  Integrated  Cl 
-HWCI 

Traceability 

•  Between  Requirements* 

-  System  to  Segment 

-  Segment  to  Cl 

-  Cl  to  Cl  (Higher  Level  to  Lower  Level) 

•  Between  Requirements  and  Design 

-  System/Segment 

-  Integrated  Cl 
-HWCI 

•  Between  Requirements  and  Test 

-  System/Segment 

-  Integrated  Cl 
-HWCI 

Problem  Reports/Action  Items/Issues 

•  Opened/Closed 

•  Reason  for  Closure 

•  Type  of  Error 

•  Severity 

•  Criticality 

•  Source 

•Age _ 


*Definition  complete  or  in  progress 
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Table  3.  Product/Process  Metrics 


Metric 

Summary  Description:  Overview  and  Purpose 

Volatility 

Indicate  changes  in  products/processes  and  reasons  for  change.  Provide  insight  into  system  maturity  and 
stability.  Aid  in  predicting  future  changes  to  products/processes  which  are  affected  by  current  changes  in 
products/processes.  Essential  in  interpreting  other  metrics,  e.g.,  progress,  traceability,  and  completeness 
metrics.  Recommended  for  requirements,  design  and  code,  and  incremental  build  definitions. 

Traceability 

Indicate  degree  to  which  development  organization  maintains  accountability  for  meeting  requirements  at  each 
life-cycle  stage  via  a  comprehensive  requirements  allocation  and  mapping  process.  Measure  relationships 
between;  (1 )  requirements  and  requirements  at  other  specification  levels,  designs,  code/databases,  builds, 
and  tests;  and  (2)  designs  and  code/databases,  builds,  and  tests.  Provide  quantitative  means  for  determining 
whether  all  required  relationships/dependencies  are  addressed.  Assist  in  exposing  incompletely  specified, 
insufficiently  analyzed,  overly  specified,  and  complex  areas  of  system.  Essential  In  interpreting  other 
metrics,  e.q.  completeness  metrics. 

Target  Resource 
Utilization 

Indicate  planned  and  actual  utilization  of  computer  resources  for  target  system.  Provide  timely  feedback  on 
whether  software  Is  being  designed  and  developed  to  fit  resources  planned  for  its  operational  use.  Assist  in 
preventing  adverse  effects  on  cost,  schedule,  and  quality  due  to  inadequate  system  sizing.  Recommended  for 
CPU.  Drimarv  memory,  mass  storage,  I/O  capacity,  and  other  applicable  resources. 

Problem  Report  and 

Action  Item 

Indicate  quality  of  products  and  process  used  to  create  them,  and  effectiveness  of  engineering  process  in 
documenting  and  addressing  problems  and  issues.  Consist  of  counts  of  problem  reports  and  action  items 
characterized  by  source,  product,  problem  ty^)e/category,  age,  severity,  criticality,  status,  and  primary  reason 
for  closure.  Recommended  for  all  products  generated  from  requirements  through  testing  and  maintenance 
activities.  Essential  in  interpreting  other  metrics. 

Size 

Indicate  magnitude  of  development  and  maintenance  effort.  Used  in  assessing  progress,  estimating 
remaining  cost  and  schedule,  identifying  technical  problems,  predicting  maintenance  cost  and  effort, 
generating  historical  data  for  future  use,  and  quantifying  the  amount  of  reuse.  Reoommended  for 
requirements,  designs,  and  code. 

For  code,  size  must  include  all  code  that  the  programmer  writes  In  any  language:  compiled/assembled 
languages,  operating  system  oommand  languages,  database  definition  languages,  graphical  user  interface 
builders,  and  expert  system  shells.  (SLOG  is  the  recommended  measurement  for  several  of  these 
languages.)  Classified  by:  physical  and  logical  statements,  statement  type,  deliverable  and  non-deliverable 
statements,  operational  and  support  statements:  and  new,  modified,  and  reused  statements. 

Complexity 

Indicate  structural  characteristics  of  software  system  logic  flow,  information  flow,  and  databases.  Useful  in 
determining  whether  work  has  been  completed  satisfactorily,  in  planning  for  code  development  and  test,  in 
identifying  technical  problems,  and  in  estimating  development,  test,  and  maintenance  cost  and  effort.  Several 
studies  have  shown  that  highly  complex  software  is  more  likely  to  contain  errors  and  is  more  difficult  to 
maintain  than  less  complex  software. 

Defect  Density 

Indicate  density''  of  product  defects  that  are  detected  during  an  Inspection  or  walkthrough.  Classified  by  type, 
criticality,  and  source.  Provide  early  insight  into  quality,  assist  in  cost/schedule  estimation,  and  indicate 
effectiveness  of  inspection/walkthrough  process.  Recommended  for  requirements,  designs,  and  code. 
Useful  in  predicting  product/process  volatility.  Essential  in  interpreting  other  metrics,  e.g.,  completeness, 
traceability,  and  yolatility  metrics. 

Fault  Density 

Indicate  density''  of  product  faults  that  are  detected  during  test  execution  or  post-test  analysis.  Classified  by 
type,  criticality,  and  source.  Indicate  effectiveness  of  testing  process.  Assist  in  determining  effectiveness  of 
other  software  processes  and  quality  of  their  products.  Recommended  for  requirements,  designs,  and  oode. 
Useful  in  predicting  product/process  volatility.  Essential  in  interpreting  other  metrics,  e.g.,  completeness, 
traceability,  test  coverage,  and  volatility  metrics. 

Test  Coverage 

Indicate  the  extent  and  adequacy  of  testing.  Assist  in  determining  test  completeness  and  test  progress.  Two 
general  types  of  test  coyerage  metrics  are  Requirements  Test  Coyerage  metrics  and  Design/Code  Test 
Coyerage  metrics.  Requirements  Test  Coverage  metrics  indicate  (a)  whether  or  not  test  documentation  is 
adequate  for  requirements  verification  and  (b)  whether  or  not  tests  fully  verify  requirements.  Design/Code 
Test  Coverage  metrics  include  counts  of  how  many  of  the  total  number  of  statements,  paths,  branches,  and 
interfaces  are  tested  and  also  upon  how  varied/realistic  are  the  input  data  sets  used  in  testing.  Together, 
these  two  tvoes  of  test  coverage  metrics  provide  data  on  product  quality  and  compliance  with  requirements. 

^Density  is  the  number  of  defects/faults  found  divided  by  the  size  of  the  product  in  which  the  defect/fault  is  detected. 
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Table  4.  Project  Resource  Metrics 


Metric 

Summary  Description:  Overview  and  Purpose 

Staffing 

Characterize  number,  discipline  (e.g.,  design,  coding,  test,  configuration  management,  quality  assurance), 
skill  level  (discipline  and  years  of  education  and  experience),  and  area(s)  of  assignment  (e.g.,  CSCIs)  for 
development  organization  personnel.  Indicate  planned  and  unplanned  changes  in  staffing  level  and 
assignments,  which  can  be  used  to  predict  whether  an  effort  is  adequately  staffed  to  preclude  adverse  effects 
on  cost,  schedule,  and  quality. 

Development,  Integration, 
and  Test  Resource 

Utilization 

Indicate  planned  and  actual  utilization  of  computer  resources  for  software  development  and  support  activities. 
Provide  timely  feedback  on  whether  planned  and  available  resources  for  each  phase  will  adequately  support 
the  activities  of  that  phase.  Assist  in  preventing  adverse  effects  on  cost,  schedule,  and  productivity  due  to 
resource  shortages.  Recommended  for  CPU,  primary  memory,  mass  storage,  I/O  capacity,  workstations, 
and  other  applicable  resources  such  as  COTS  software. 

Table  5.  Progress  Metrics 

Metric 

Summary  Description:  Overview  and  Purpose 

Completeness 

Indicate  work  accomplished  versus  work  remaining  in  requirements  and  design  specification,  coding, 
inspection,  unit  test,  integration  and  test,  and  system  test.  Assist  in  estimating  cost  and  schedule  remaining, 
in  identifying  technical  problem  areas,  and  in  determining  readiness  to  proceed  to  the  next  phase.  Each  class 
of  completeness  indicator  (where  a  class  focuses  on  a  single  product,  e.g.,  requirements,  design,  code,  or 
test)  should  be  used  in  conjunction  with  the  other  measures  for  that  class  as  indicated  in  the  "Integrated 
Prooress"  metric  description  below. 

Integrated  Progress 

Indicate  overall  progress  in  requirements,  design,  code,  and  test.  Encompass  measures  of  completeness, 
volatility,  traceability,  defect  and  fault  density,  problem  reports/action  items,  and  test  coverage  as  appropriate 
for  phase  and  product  under  consideration.  (The  high  information  content  of  Integrated  Progress  metrics 
lends  insiqht  into  progress  that  measures  of  completeness  alone  cannot  supply.) 
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